add io offload prototype #1774

rem1776 · 2025-09-22T17:59:14Z

Description
draft for the io offloading changes

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Please also note
any relevant details for your test configuration (e.g. compiler, OS). Include
enough information so someone can reproduce your tests.

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules
New check tests, if applicable, are included
make distcheck passes

… test

…resses

vithikashah001

needs documentation for variables, subroutines and functions.

fms2_io/netcdf_io.F90

offloading/metadata_transfer.F90

uramirez8707

Some minor changes, but I think this is a good start

offloading/offloading_io.F90

test_fms/offloading/Makefile.am

uramirez8707 · 2026-01-08T18:52:01Z

offloading/offloading_io.F90

+  integer, parameter :: non_domain_decomposed = 1 !< enumeration for type of domain
+  integer, parameter :: Unstructured_grid = 2 !< enumeration for type of domain
+
+  integer, parameter :: max_files = 10 !< amount of offloaded files to allocate space for


TODO Make this a namelist, runtime constant, etc

added with 4a5ed59

uramirez8707 · 2026-01-08T19:14:06Z

offloading/offloading_io.F90

+  !! Registers an axis to a netcdf file on offloaded PEs. File must have been opened with open_file_offload.
+  interface register_axis_offload
+    procedure :: register_netcdf_axis_offload
+    procedure :: register_domain_axis_offload


TODO we need a register_unstructured_axis_offload for the unstructured grid

added any leftover TODOs as comments with 969513e

uramirez8707 · 2026-01-08T19:17:19Z

offloading/offloading_io.F90

+
+      !< The number of tiles must be the same as the number of offloading pes
+      !if (size(pe_out) .ne. ntile ) &
+      !  call mpp_error(FATAL, "The number of offloading PEs must be the same as the number of tiles of the domain")


This should be added back, and it should crash if the ntiles is not a multiple of the ntiles

added with 969513e

uramirez8707 · 2026-01-08T19:21:09Z

offloading/offloading_io.F90

+      call mpp_broadcast(filename_out, str_len, pe_in(1))
+      call mpp_broadcast(global_domain_size, size(global_domain_size), pe_in(1))
+      call mpp_broadcast(ntile, pe_in(1))


TODO maybe we can find a way to do this all in one mpp_brodcast call (or mpp_send/mpp_receive)

uramirez8707 · 2026-01-08T19:32:53Z

offloading/offloading_io.F90

+    integer, allocatable :: all_current_pes(:)
+    integer, allocatable :: broadcasting_pes(:)
+    logical :: is_model_pe
+    character(len=255) :: var_info(2)


Update to use the max str length variable

uramirez8707 · 2026-01-08T19:34:16Z

offloading/offloading_io.F90

+    if (.not. is_model_pe) then
+      select type(wut=>this%fileobj)
+        type is(FmsNetcdfDomainFile_t)
+          call register_axis(wut, trim(var_info(1)), trim(var_info(2)))
+      end select
+    endif


Let's clean this up

#1774 (comment)

uramirez8707 · 2026-01-08T19:36:44Z

offloading/offloading_io.F90

+    type(domain2D) :: domain_out
+    type(domain2D) :: domain_in
+    integer :: isc, iec, jsc, jec, nz, redistribute_clock
+    character(len=255) :: varname_tmp(1)


use max str length

J-Lentz · 2026-01-08T19:32:29Z

offloading/offloading_io.F90

+  !> Structure to hold offloading file information
+  type :: offloading_obj_out
+    integer :: id
+    character(len=:), allocatable :: filename !< filename of the offloaded netcdf file


Is there a need to store the filename here, as opposed to getting it from fileobj%path?

No looks like it can be removed, I'll add that change.

J-Lentz · 2026-01-08T19:36:28Z

offloading/offloading_io.F90

+    character(len=:), allocatable :: filename !< filename of the offloaded netcdf file
+    class(FmsNetcdfFile_t), allocatable :: fileobj !< fms2_io file object
+    type(domain2D) :: domain_out !< domain on offloading PEs
+    integer :: type_of_domain !< type of domain (domain_decomposed, non_domain_decomposed, Unstructured_grid)


It doesn't seem that type_of_domain is used. Is it needed?

Should be able to remove this as well.

J-Lentz · 2026-01-08T19:37:45Z

offloading/offloading_io.F90

+    integer :: type_of_domain !< type of domain (domain_decomposed, non_domain_decomposed, Unstructured_grid)
+  end type
+
+  !> Offload equivalent of open_file in fms2_io_mod


This comment seems like it belongs above open_file_offload.

J-Lentz · 2026-01-08T19:39:13Z

offloading/offloading_io.F90

+      type_of_domain = domain_decomposed
+    type is (FmsNetcdfUnstructuredDomainFile_t)
+      type_of_domain = Unstructured_grid
+    end select


type_of_domain is assigned, but never used.

J-Lentz · 2026-01-08T19:45:53Z

offloading/offloading_io.F90

+    endif
+
+    if (.not. is_model_pe) then
+      select type(wut=>this%fileobj)


Since FmsNetcdfDomainFile_t has no types derived from it, perhaps change fileobj from class(FmsNetcdfDomainFile_t) to type(FmsNetcdfDomainFile_t), and remove the select type.

@uramirez8707 had a similar comment too I believe.

Since the main derived type (offloading_obj_out) uses a class(FmsNetcdfFile_t), i was thinking it might be cleaner to just make all the routines take class(FmsNetcdfFile_t) instead and just error out for now in case its not a supported file type.

I figured that makes it a bit easier to use and will give us some flexibility in the future, but lmk what you guys think.

J-Lentz · 2026-01-08T19:47:02Z

offloading/offloading_io.F90

+    endif
+
+    if (.not. is_model_pe) then
+      select type(wut=>this%fileobj)


See above comment, regarding class(FmsNetcdfDomainFile_t) to type(FmsNetcdfDomainFile_t).

J-Lentz · 2026-01-08T19:51:25Z

offloading/offloading_io.F90

+      select type(wut=>this%fileobj)
+        type is(FmsNetcdfDomainFile_t)
+          call register_field(wut, trim(var_info(1)), trim(var_info(2)), var_info(3:))
+      end select


This doesn't seem to handle the case where fileobj is FmsNetcdfFile_t as opposed to FmsNetcdfDomainFile_t - is this case supported?

J-Lentz · 2026-01-08T19:59:08Z

offloading/offloading_io.F90

+  !! This routine should be called from both the model PEs and the offload PEs, with the full list
+  !! of pes for each group being provided. The model PEs will broadcast the filename and domain.
+  subroutine open_file_offload(fileobj, filename, domain_in, pe_in, pe_out)
+    class(FmsNetcdfFile_t), intent(inout) :: fileobj !< fms2_io file object


Some other subroutines (e.g., write_data_offload) only seem to support FmsNetcdfDomainFile_t and not the FmsNetcdfFile_t base type. If non-domain-decomposed files are unsupported, type(FmsNetcdfDomainFile_t) should probably be used consistently for fileobj arguments.

rem1776 · 2026-01-12T17:50:54Z

@uramirez8707 @J-Lentz @vithikashah001 This should be ready for another pass at reviews, I think every comment should be covered. See #1774 (comment) for how i handled the typing, i just made everything take the base FmsNetcdfFile_type class for now with some type checking added.

uramirez8707

Looks good for now, but we have a lot of clean up to do.

Let’s plan the next steps next week.

uramirez8707 and others added 23 commits November 3, 2023 11:59

#NOTHING_WORKS

f6ebba0

wut

972bc90

push uncommitted updates

06d2405

separate and move modules to new offloading directory

bdde8ed

Merge remote-tracking branch 'origin/main' into HEAD

130ab90

add routines for metadata transfer type

90c0c8c

add constants for allocatable workaround and expand metadata transfer…

710d887

… test

add inherited class structure for reals and ints

f762986

finish up support for all data types and add to test

fbd739e

add metadata object usage to offload moduld

757764b

free memory after redistribute calls and adjust domains total mem add…

1fe352f

…resses

fix for cubic domains

b3770af

add clocks for offload writes/redistributes

0d54864

update for layouts and fix for more than 1 offload pe per tile

9e31d77

fix bug with different layouts and pe counts

6c56515

Merge branch 'main' of github.com:noaa-gfdl/fms into metadata-transfer

1176605

remove attribute_type from base class and any unused variables/args

ed4c178

comments and linter

890bfd6

undo accidental change to data override test

9541c66

add cast for r8 build option to pass

79a3af4

add ifdefs for for mpi-less build

a9ce72c

fix mpi transfer for intel but not gcc :(

18c7178

fix mpi struct bug with gcc and remove mpp test

ecba40d

rem1776 marked this pull request as ready for review December 1, 2025 17:52

rem1776 requested review from bensonr, uramirez8707 and vithikashah001 as code owners December 1, 2025 17:52

Merge branch 'main' into metadata-transfer

b2be463

vithikashah001 requested changes Dec 29, 2025

View reviewed changes

fms2_io/netcdf_io.F90 Show resolved Hide resolved

fms2_io/netcdf_io.F90 Show resolved Hide resolved

offloading/metadata_transfer.F90 Outdated Show resolved Hide resolved

add more documentation and any missing headers

c0ab0dc

J-Lentz self-requested a review January 5, 2026 18:50

fix typo and white space

08a7da9

uramirez8707 requested changes Jan 8, 2026

View reviewed changes

J-Lentz requested changes Jan 8, 2026

View reviewed changes

rem1776 added 3 commits January 9, 2026 14:36

add any missing apache licenses

596da47

add nml for io offload module

4a5ed59

add in todo comments and some type/length changes from reviews

969513e

rem1776 force-pushed the metadata-transfer branch from 80b7ef9 to 969513e Compare January 12, 2026 15:50

rem1776 added 2 commits January 12, 2026 12:32

clean up typing for netcdf file objs

07ea5a8

line lengths for linter

9dc1d4f

J-Lentz self-requested a review January 12, 2026 20:13

J-Lentz approved these changes Jan 12, 2026

View reviewed changes

vithikashah001 requested review from uramirez8707 and vithikashah001 January 12, 2026 20:31

vithikashah001 approved these changes Jan 12, 2026

View reviewed changes

uramirez8707 approved these changes Jan 12, 2026

View reviewed changes

vithikashah001 merged commit 3d4ac8c into NOAA-GFDL:main Jan 13, 2026
28 checks passed

add io offload prototype #1774

add io offload prototype #1774

Uh oh!

Conversation

rem1776 commented Sep 22, 2025

Uh oh!

vithikashah001 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uramirez8707 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rem1776 commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uramirez8707 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rem1776 commented Jan 12, 2026 •

edited

Loading